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ABSTRACT 


IL  SYSTEM  EFFECTIVENESS  ANALYSIS 


A  methodology  for  evaluating  a  large  scale  C3  system  is 
developed.  The  first  step  of  this  methodology  consists  of  a 
procedure  for  evaluating  a  system  using  a  simple  model  of  it 
The  second  step  consists  of  an  algorithm  aimed  at  determining 
the  smallest  number  of  experiments  that  will  enable  one  to 
evaluate  the  effectiveness  of  an  actual  C3  system  or  testbed. 

This  methodology  is  applied  to  an  abstracted  version  of  an  actual 
air  defense  system:  a  mathematical  model  of  this  system  is 
pfTffwwd  and  the  system  is  evaluated  on  the  basis  of  this  model. 
The  experiment  design  procedure  is  assumed  to  have  been 
implemented  and  data  produced:  it  is  then  shown  how  the  actual 
system  would  have  been  evaluated  based  on  these  data. 

I.  INTRODUCTION 


A  major  focus  of  the  research  in  Command  and  Control 
(C2)  is  the  need  to  assess  quantitatively  the  utility  of  Command. 
Contra,  and  Communications  (C3>  systems.  Over  the  past 
decade,  methodologies  have  been  proposed  that  provide  system 
developers  with  powerful  tools  for  evaluating  the  effectiveness 
of  systems  [Mishan  1976.  Dersin  and  Levis  1981.  White  1985). 
All  these  methodologies  assume  that  experimental  data  can  be 
gathered  for  the  system  to  be  evaluated. 


A  new  problem  arises  when  very  large  scale  economic, 
inr»»i  or  military  systems  are  considered  because  it  is  often  not 
possible  to  run  a  large  number  of  experiments  and  collect  all  the 
necessary  to  cany  out  the  assessment.  A  common  approach 
to  analyse  and  evaluate  system  design  is  to  build  a  test  bed 
which  provides  the  developer  with  the  ability  to  consider  many 
different  configurations  of  the  same  system  and  with  a  means  to 
gather  data  on  these  configurations  in  order  to  evaluate  them. 
However,  for  each  configuration  of  the  system,  one  roust  design 
experiments  to  run  in  order  to  obtain  data. 


In  response  to  this  need,  a  methodology  is  presented  in  this 
paper  it  aims  at  die  design  of  experiments  to  run  on  a  system  so 
that  the  effect! ven ness  of  any  configuration  can  be  evaluated. 
This  methodology  draws  on  the  framework  first  developed  by 
Dersin  and  Levis  (1981).  and  then  applied  to  C3  systems  by 

Bouthonnicr  (1984),  and  Cothier  (1986).  The  method  of 
analysis  used  is  based  on  relating  the  performance  of  a  system  to 
the  requirements  of  the  mission  u  has  to  fulfill. 


•This  work  was  conducted  at  the  MIT  Laboratory  for 
Information  and  Decision  Systems  with  support  provided  by  the 
French  Delegation  Generate  pour  l'Armement  and  with  partial 
support  provided  bv  the  Joint  Directors  of  Laboratories  through 
the  Office  of  Kaval  Research  under  Contract  No. 
N0001 4-I5-K-0782. 


Let  us  consider  a  set  U  which  represents  the  universe:  U 
will  be  called  the  universal  set  It  may  contain  a  great  diversity 
of  elements  such  as  physical  entities,  data  bases,  or  doctrines.  A 

foal  is  defined  to  be  a  particular  desired  state  of  the  universe  U. 

ypical  goals  are :  to  produce  data,  to  transmit  information,  or 
to  defend  one's  assets.  A  tysttm  S  is  defined  to  be  a  set  of 
elements  of  the  universe  U  that  act  together  by  exchanging 
information  (connectivity)  toward  the  achievement  of  a  particular 
goof.  The  set  S  is  a  subset  of  U.  An  element  u  of  the  universe  is 
included  in  the  snnronmtnt  E  if  and  only  if  u  does  not  belong  to 
S  and  the  system  can  act  upon  u,  and  u  can  act  upon  the  system. 
The  context  C  then  is  defined  as  the  complement  of  the  set 

S  u  E  in  the  universe  Cu"  denotes  the  union  of  two  sets,  and 

W  their  intersection).  With  these  three  definitions  one  can 
easily  deduce  die  following  properties: 


U-SuEuC,  SnE-e.  S n  *•  e.  E n C « #  (1) 


The  term  mission  designates  the  task  the  system  has  to 
perform.  It  is  the  particular  state  of  the  environment  that  has  to 
be  achieved  by  the  system.  The  mission  depends  on  the  system 
considered:  for  instance,  the  mission  of  a  C3  system  is  to 
provide  "adequate”  information  to  effectors  on  the  basis  of  the 
data  it  receives  from  the  tensors. 

Foramsters  are  the  independent  variables  in  system 
effectiveness  analysis.  The  system  parameters  are  entities  whose 
value  determine  system  behavior  and  specify  system  structure. 
They  are  defined  within  the  system  boundary.  Environment  or 
context  parameters  refer  to  the  independent  variables  that 
describe  the  environment  or  the  context 


Utosunt  of  Forformonct  (MOPs)  are  measurable 
quantities  that  describe  system  properties  or  attributes.  Their 
values  depend  on  the  values  of  the  parameters  that  characterize 
the  system,  the  environment,  and  the  context.  The  system 
measures  of  performance  vary  in  Q,  a  subset  of  Rn  where  n  is 
the  number  of  MOPs.  Since  the  parameters  are  constrained  to  be 
in  a  subset  P  of  RP  where  p  is  the  number  of  parameters,  one 
cmmc  expect  the  MOPS  so  oka  any  value  in  ft.  Since  each  MOl 
is  a  function  of  several  parameters,  one  can  define  a  mapping 
from  the  parameter  space  huo  the  MOP  ^acc.  This  mapping  it 
obtained  by  exarcising  the  system  (or  by  running  nmuUuons) 
for  different  contexts  and  different  values  of  the  system 
panmeien  in  order  to  mmmm  the  reachable  MOP  values.  The 
set  of  das  reachable  MOP  values  k  dre  system  locus  L,  (Fig  1) 
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Fig.l  System  Locus 

The  mission  the  system  has  to  achieve  is  defined  by 
requirements  in  the  MOP  space.  These  requirements  are 
obtained  by  running  models,  games  or  plans  for  different 
contexts  and  for  different  mission  parameters.  In  order  to 
enable  one  to  compare  the  mission  and  the  system,  the 
requirements  must  be  expressed  in  term  of  the  MOPs  defined  for 
the  system:  the  mission  MOPs  (expressing  the  mission 
requirements)  must  be  the  same  as  the  system  MOPs 
(expressing  the  system  capabilities).  The  set  of  MOP  values  that 
unify  the  mission  requirement!  constitutes  the  mission  locus 
Lg,  (Fig.2). 


Some  Mtosuret  of  Efltctivontu  (MOEs)  can  be  derived 
from  the  comparison  of  the  two  loci  Lt  and  L^.  Qualitatively, 

the  greater  the  intersection  of  die  two  loci .  the  more  effective  the 
system  is.  If  V(L)  is  a  measure  on  the  locus  L  (Fig. 3),  one  can 
define  the  following  MOEs: 

Ej  »  V(LS  n  l^WfL,)  (2) 

E2.V(L,nLlnW(Ltn)  (3) 

where  Ej  is  the  degree  to  which  the  system  capabilities  are 

included  in  the  mission  locus  (it  measures  how  well  the  system 
capabilities  are  used  for  the  mission  considered)  and  Ej  is  the 
degree  to  which  the  mission  locus  is  covered  by  the  system  (it  is 
the  degree  of  coverage  of  the  mission  by  the  system  ) 

The  important  fact  in  the  passage  from  an  MOP  to  an  MOE 
is  the  consideration  of  requirements  In  the  absence  of 
requirements,  the  system  locus  cannot  tell  how  effecuve  the 
system  actually  is 


Fig.3:  Measures  of  effectiveness 

Fora  given  system,  one  can  define  many  different  MOEs 
representing  effectivenesses  from  various  standpoints.  These 
MOEs  are  called  penial  MOEs.  Let  E^-.Ep  be  the  partial 

MOEs  for  a  given  system  Defarau  (1961)  has  town  that,  under 
certain  conditions,  there  exists  a  real  valued  function,  a  utility 
function,  which  is  continuously  dependent  on  the  Ej.  Let  U  be 
such  function  taking  values  between  0  and  1: 

[0.1JP  - - >10,1] 

Ej f 2**"*^p  ’ “ 1 . 'P  U(  E j JEj-.-Ep) 

then,  the  global  measure  of  effectiveness  can  be  taken  to  be 

UfEjJj.-.^p). 

The  System  Effectiveness  Analysis  methodology  presented 
in  this  section  is  mad  now  id  design  experiments  to  run  on  large 
seek  tysteme  so  that  these  rywemr  may  be  improved 

m.  EXPERIMENT  DESIGN  PO*  LARGE  SCALE  SYSTEMS 

To  determinr  the  system  locus  of  •  large  scale  system 
requires,  in  general,  operating  the  system  at  an  extremely  large 
number  of  sets  of  parameter  values.  The  procedure  presented  in 
this  section  provides  a  means  far  deiermiiitng  only  a  ratal]  set  of 
experiments  to  be  run  on  the  system:  the  tusulting  mall  number 
of  experimental  values  will  he  combined  with  the  results 
obuuaad  from  e  amplified  aradieraeticel  seeds!  of  the  system  to 
construct  the  system  locus  of  the  actual  lystem. 

A  simplified  ttihamatirel  model  of  the  actual  system  is 
fhn  rowtidtreit  this  model  can  be  ranmeamad  by  a  mapping  T 
from  the  parameter  apace  into  the  MOP  apace.  Two  mapping! . 
and  consequently  two  loci  can  he  centiderad.  Fine  using  the 
mathematical  model  T.  tin  pnranmar  ipuce  enn  be  aaapped  into 
the  "model'  system  locus.  Second,  dm  actual  system,  if  it  could 
be  exercised  for  el)  the  values  of  the  parameter  apace,  would 
yield  the  “actual"  system  locus  (Ftg.4).  Since  the  model  it  a 
simplified  one,  the  node!  locus  is  a  rough  approximation  of  the 
actual  locus 

actMl  incut  am)  uk.i 


raraaetei  Spec* 


MV  Space 


Fig  4  Actual  and  Model  System  Loci 


Since  the  objecti ve  it  to  determine  the  actual  locus,  the  key 
idea  is  to  obtain  die  model  locus,  and  a  few  points  of  the  actual 
locus,  and  to  determine  a  mapping  T"  that  oansfarms  the  model 
locus  into  the  actual  locus  in  the  MOP  space.  Witht  this  raping, 
the  actual  locus  can  be  obtained  indirectly:  if  "A  is  the  actual 
mapping  from  the  parameter  space  into  the  MOP  space,  then 
A-Tof,  where  V  denotes  the  composition  of  two  functions 
(Fig J).  With  this  algorithm,  only  a  few  points  belonging  to  the 
actual  locus,  and  therefore  only  a  few  experiment!  will  be 

necessary  to  evaluate  the  locus  of  the  actual  system  and 
eventually  the  effectiveness  of  this  system.  On  Fig. 5  the  actual 
mapping  A  is  denoted  (A  *  Tof)  to  emphasize  the  fact  that  it 
cannot  be  obtained  directly  but  only  as  a  composition  of  f  and  T. 
Since  f  is  assumed  to  be  known,  the  focus  of  the  remaining  pan 
of  this  section  is  the  determination  of  T. 


I.  (Actual  Locus) 

aa 

Fig.  5:  Determination  of  the  actual  mapping 


A  Inversion  algorithm 

This  sub-section  provides  a  brief  description  of  an 
inversion  algorithm  The  purpose  of  this  algorithm  is  as  follow 
for  a  given  value  in  the  MOP  space,  find  a  value  in  the  parameter 
space  that  yields  through  T  the  desired  value  in  the  MOP 
space.  The  algorithm  must  determine  whether  T  can  be 
inverted  at  a  specific  MOP  value,  and  if  it  can,  the  algorithm  will 
yield  •  parameter  vector  (which  may  not  be  unique) 
corresponding  to  the  desired  MOP  value 

A  system  with  m  MOPs  (Xjli-1 . m)  and  n  parameters 

(Pjij-1 . n)  is  considered,  generally,  n  and  ro  are  not  equal.  If 

T  is  the  mapping  from  the  parameter  space  into  the  MOP 
space  then  g»Agt).  where  &  and  p  are  column  vectors  If  jjq  is 
an  initial  combination  of  parameters,  the  corresponding  point  m 
the  MOP  space  is  iQ-ftpo),  Bo  Wll>  **  CAUcd  1  <teuc  opening 
point'.  One  is  interested  in  reaching  a  desired  MOP  value  g^, 

with  ■  Aq  ♦  fig  :  one  has  to  find  fig  such  that  f(  BO  *  fa  )  ■ 
*d 

The  Ant  step  is  so  determine  a  small  variation  fig  wound  gQ 

companding  to  a  small  desired  variation  fig  around  |q.  Since  f 
is  generally  non -invertible,  one  has  to  And  an  algorithm  that 

determines  fig  In  order  to  carry  out  this  Ant  step,  f  is  assumed 
to  be  differentiable  »>  t  ■  JLo  .  and  a  singular  value 
decomposition  of  a  linear  approximation  of  T  around  is 
usad  (Martin  1916]  The  algorithm  provides  a  means  to 

determine  whether  a  small  variation  ftp  can  be  found,  and  if  it 


can,  a  means  far  choosing  among  many  possible  fig .  Moreover, 
the  algorithm  is  designed  in  such  a  manner  that  it  guarantees 

Bptfig  to  be  included  in  the  set  of  allowable  parameter  values  of 

the  system  (parameter  locus):  therefore  a  real  experiment  can  be 
conducted  at  this  parameter  value  [Martin  1986]. 

Since  the  single  stage  algorithm  generally  yields  an 
approximation  of  the  MOP  value  one  wants  to  reach,  a  second 
step  in  the  procedure  is  to  iterate  the  single  sage  algorithm  until 
the  difference  between  the  desired  value  and  the  computed  value 
is  small  enough  for  the  application  at  hand.  Therefore,  this 
inversion  algorithm  provides  a  means  to  find  a  parameter  value 
such  that  the  model  T  when  applied  to 

this  parameter  value  will  yield  a  point  in  the  MOP  space 
arbinrily  doae  to  a  desired  MW  value. 

B.  Experimental  design 

After  applying  the  inversion  algorithm,  a  vector  p  has  been 
obtained  such  that  g<j«f(g);  the  error  io  the  demrmination  of  p  is 

assumed  to  be  so  small  as  to  be  negligible.  To  determine  what 
the  actual  system  locus  looks  like,  we  will  choose  a  small 
number  of  desired  points  in  the  MOP  space  (g<j).  Far  each  of 
these  points,  with  the  inversion  algorithm,  we  will  find  a 
parameter  vector  such  that  •  f(p).  Then,  an  experiment  will 

be  ran  on  the  actual  sysKin  at  this  parameter  vector  ,  the  outcome 
of  the  experiment  will  be  a  pointy  indie  MOP  apace.  Since  the 
simplified  model  and  the  actual  system  are  close  but  not  equal, 
the  values  g,  and  gj  are  going  to  be  different  (Fig.6). 


Fig  6  Mathematical  and  Experimental  MOP  values 


Wish  this  procedve.  for  each  desired  point  selected  in  the 
oaodel  locus,  there  will  be  a  corresponding  point  in  the  actus: 
locus;  so  a  given  set  of  points  in  the  model  locus  will  correspond 
a  set  of  points  in  fee  actual  locus  The  only  available  informs aon 
for  aeiecang  the  points  g^  is  the  system  locos  obtained  from  the 
ratheraocal  oaodel  With  the  rapping  f,  one  can  easily 
determine  this  locus.  Since  one  is  untreated  in  having  a 
ealecbon  of  points  that  represents  the  —are  locus  aa  opposed  to 
soma  pan  or  regtoo.  the  points  that  will  be  chosen  must  be 
dsaoihi^allorer  the  loctts  A  simple  way  to  choose  a  small 
*  poiau  thes  are  represantaovt  of  the  locus  is  to 
inscribe  « in  so  a  hnt.an.tl  rectangular  paraUilcpiped.  and  to 
choose  ftctangeaey  points,  the  two  dunenrioruU  cast  is  shown 
tn  Fig. 7.  The  procedure  is  given  in  detail  in  [Martin  1916] 


This  method  for  choosing  the  points  (14)  in  the  model 

locus  is  not  only  simple;  it  is  the  one  that  allows  one  to  select  the 
minimum  number  01  points  for  looking  at  the  whole  locus  ( 2n 
points  will  be  selected  where  n  is  the  dimension  of  the  MOP 
space)  [Martin  1986]. 

C  Reconstitution  of  the  actual  system  locus 

We  now  have  all  the  tools  required  to  select  a  small  number 
of  points  in  the  actual  locus,  and  to  compute  the  mapping  T  that 
transforms  the  model  locus  into  the  actual  locus,  First,  let  us 
determine  the  small  number  of  points  we  are  looking  for  in  the 
actual  locus:  We  apply  the  inversion  algorithm  to  the  r  ■  2n 
points  (ggj)  i— l..,.jr  selected;  it  yields  r  vectors  (pj)  i-l,._/  in 
the  parameter  space.  Then,  experiments  corresponding  to  these 
parameter  vectors  are  run  on  the  actual  system:  that  it,  the 
experimental  conditions  are  set  as  required  by  the  parameter 
vectors.  This  is  always  feasible  because  the  parameter  vectors 
determined  with  the  inversion  algorithm  are  constrained  to 
belong  to  the  set  of  admissible  parameters  [Martin  1986].  As 
shown  above,  the  procedure  for  choosing  the  points  g^j  is  the 

one  that  allows  one  to  look  at  the  whole  locus  with  the  minimun 
number  of  points:  thus,  given  the  con  train!  that  one  must  look  at 
the  whole  locus,  the  minimum  number  of  experiments  are 
act  Italy  run.  The  outcome  of  these  experiments  will  be  r 
experimental  values  (g^)  i»l,..„r  in  the  MOP  space  (Fig.7). 


Kfij  )  .  !„•  tip«ri»*nt<p.1  )  ,j.j  < 

Fig  7  Experimental  results 

We  assume  that  the  transfarmanon  T  is  the  composition  of 
a  translation  ^  and  a  linear  transformation  L 

Ti  -  La  *  Y  (4) 

Let 

e,  -  ILta  ♦  Y  ■  Itx  ■  «> 

where  1:  II  represents  the  euclidian  norm  Then  the  trant- 
formation  we  art  searching  is  the  one  that  minimizes  the 
following  expression  (r  is  the  number  of  experiments! 

Jm<l  *<2*  *et  (6) 

With  this  transformation  T,  for  each  point  g^  m  the  model 
locus,  the  corresponding  point  in  the  actual  locus  is 

g,  “T  (in'  (7) 


We  can  interpret  A  at  the  transformation  Tof  that  maps  the 
parameter  locus  into  the  actual  system  locus.  This  mapping  is 
the  outcome  of  a  mathematical  model  combined  with  the  snail 
number  of  experimental  values  that  one  can  usually  afford  to  run 
on  the  actual  system.  It  should  be  noted  at  that  point  that,  if  the 
actual  system  could  be  exercised  for  all  the  values  of  the 
parameter  locus,  it  would  probably  yield  an  actual  system  locus 
slightly  different  from  the  one  obtained  at  the  end  of  this  section; 
given  the  experimental  constraints,  the  actual  locus  obtained 
with  A«Tof  is  the  best  approximation  of  the  actual  system  locus 
that  would  be  obtained  by  running  as  many  experiments  as  one 
would  like.  Then,  with  die  actual  system  loots,  one  can  evaluate 
the  effectiveness  of  the  actual  system,  as  opposed  10  evaluating 
the  effectiveness  of  a  simplified  model  of  the  system. 

The  steps  of  the  experiment  design  procedure  and  their 
interrelationships  are  shown  in  Fig.8. 


Fig  I  Steps  of  the  experiment  design  procedure 


rv.  AN  AIR  DEFENSE  SYSTEM 

In  this  section,  s  Isrgs  scale  system  is  presented;  it  will  br 
used  ss  an  example  to  illustrate  the  SEA  methodology  and  the 
experiment  design  process  Thu  illustration  will  be  provided  b\ 
a  military  air  defense  system  known  as  'Idennftcauon  Friend 
Foe  Neutral'  (1FFN) designed  for  the  central  region  of  Europe 
The  overall  mission  of  the  system  is  10  defend  a  specified 
airspace  from  an  air  attack  earned  out  by  enemy  aircraft  and 
missiles 

This  system  is  shown  in  Fig.R,  it  is  composed  of  nodes 
that  coordinate  the  action  of  weapons,  these  nodes  can  be 
classified  into  three  wmja  groups:  ftie  Dvecung  Canters  (FDCi. 
Control  Reporting  Centers  (CRC).  and  a  dwd  group 
of  more  specific  nodes  such  at  da  is  bean  and  higher  level 


nodes.  Their  role  is  to  coordinate  the  weapons  shown  in  Fig.9; 
these  weaponi  can  be  divided  into  two  categories:  Surface  to  Air 

Missiles  (SAM)  such  u  Hawks  and  Patriots  units,  and  fighter 
planes  such  as  F-1S  and  F-16  aircraft.  The  mission  of  the 
system  is  to  engage  and  destroy  hostile  airborne  targets  or 
otherwise  deny  the  enemy  access  to  the  defended  airspace:  in 
particular,  the  enemy  aircraft  must  be  stopped  before  they  can 
fire  missiles  at  friendly  assets.  The  system  must  be  selective 
enough  to  minimize  killing  friends  (F-1S  and  F-16)  or  neutrals 
such  as  commercial  aircraft  that  are  assumed  to  be  flying  in  the 
Genual  Region  at  the  time  of  the  battle. 


In  this  model,  the 
of  a  straight  line  (FS< 
separating  the  friendly 
consideration  lies  behind  the  FSCL,  and  hostile  aircraft  arc 
heading  towards  this  line  at  speed  V.  This  model  is  represented 
in  Fig.10: 


I  fttrlfri  1 

Ftilnot.  _ 

M«we 

1  1 

rv 

U=h 

Fig.9  Structure  of  the  actual  IFFN  system 


NE-3A:  NATO  airborne  early  warning  tytem.  this  is  a  high 
altitude  detection  aircraft 

SIS:  Special  Information  System:  this  is  a  source  of  intelligence 
information  available  to  basic  nodes  of  the  system  (CRC  or 
Control  and  Reporting  Center  nodes). 

CRC:  Control  and  Reporting  Center,  this  C~  node  is  responsible 

for  die  overall  coordination  of  the  synem 

FDC:  Fire  Directing  Center,  these  C2  nodes  arc  responsible  for 

the  coordination  of  a  battalion 

FU:  Fmng  Unit  BN.  Battalion  BDE:  Brigade 

In  this  complex  system,  the  missiles  fired  cither  by  a 
fighter  or  by  a  SAM  unit  are  Beyond  Visual  Range  (BVR) 
weapons:  a  finng  unit  does  not  see  the  targets  it  is  shooting  at, 
the  fire  parameters  are  given  to  this  finng  unit  by  the  C2  system 
on  the  basts  of  identification  performed  by  other  units  called 
'detecting  units'  Thu  indirect  identification  process  justifies  the 
C2  structure  that  bet  above  the  weapons  in  Fig  9;  it  is  the 
responsibility  of  this  c’  system  to  pass  correct  and  accurate 
parameters  to  finng  units 


I  of  this 


B  Sunplifiad  model  an  overview 

In  the  simplified  model,  the  enemy  forces  are  enurned  to 
consist  of  aircraft  only,  these  enemy  aircraft  seek  to  eater  the 

Meads  tamtory.  they  can  fire  Air  to  Surface  Missiles  (ASM) 
and  An  to  Air  Missiles  (AAMj  in  order  to  destroy  both  pound 
■nits  and  airborne  units  For  their  defense,  the  friends  have 
ancraft  that  can  fire  A  AM,  and  ground  unm  that  can  fire  Swfaoc 
to  An  Missiles  (SAM)  An  enemy  unit  will  refer  to  aircraft,  a 
friendly  asset  will  refer  to  both  aircraft  and  ground  finng  units, 
for  the  neutrals,  a  unit  will  refer  to  an  aircraft  (commercial 


In  the  neat  sub-secuons.  e  simplified 
dex  system  u  introduced 
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Fig.10  Simplified  IFFN  model 


ahnne  of  die  system 
enty  i  air  surface  missiles 

inf  to  enemy's  aircraft  from  the  FSCL 
s  aircraft 


An  aircraft  will  be  detected  by  a  given  detecting  unit  and 
engaged  by  another  unit  (engaging  unit):  the  task  of  the  C3 
system  is  m  identify  correctly  an  aircraft  and  »  allocate  it  to  a 
given  engaging  unit.  In  order  to  protect  friendly  esseu.  the 
enemy's  aircraft  meat  be  stopped  before  they  reach  Rj.  and  can 


fire  missiles.  It  is  assumed  that  the  friendly  aircraft  as  well  as 
the  neumlasrcnft  am  flying  at  any  speed  and  in  any  dweetion  in 
the  diagram  sketched  above.  In  whet  follows,  this  simplified 
modal  is  developed,  bated  on  the  work  carried  out  by  Log  icon 
(Logicon  19S6);  throughout  the  section,  the  nouoon  defined  in 
the  IFFN  dnretnau  (IFFN  Teat  Flan  1915,  Logicon  1916]  will 
be  used 

C.  Parameter  definition 


To  carry  out  the  effectiveness  analysis  of  this  system,  one 
must  first  identify  the  parameters  (the  independent  variables) 
The  relevant  parameters  are  those  denned  in  the  IFFN 
documentation  (IFFN  Tan  Pirn  19*5.  Logicon  19*6).  They  arc 
indepaadeat  variables  included  in  (0,1].  For  tbit  analysis,  the 
ptnmmn  will  be 


this  fact  is 


lad  to  pan  information  between  two  nodes  fPj ) 

bather  tin  SIS  (Special  Information  System)  is 
into  tin  CRC  (Consol  and  Reporting  Center), 
dad  by  a  vmying  tin  time  delay  raqtrirad  ro  pass 


•  range  from  atienft  to  PSO.  at  time  of  detection  (P2) 
responds  to  tits  variable  ACF  (Air  Congo)  Procedure),  the 
i  or  varying  the  ACF  is  at  turned  to  be  the  vanaoon  of  the 
t  from  aircraft  to  FSCL  at  tuna  of  drracnon:  the  better  the 
,  the  larger  ties  range 


-  quality  of  identification  (P3). 

•  level  of  oennlizadon  of  control  (P4) 

•  quality  of  target  allocation  and  engagement  (Pj): 

it  corresponds  to  the  quality  of  the  Q&A  IFF  devices  (Question 
and  Answer  devices  for  Identification  Friend  Foe):  since  Q&A 
IFF  devices  provide  local  ID  information  at  the  weapon  level, 
the  quality  or  these  components  has  a  direct  effect  on  allocation 
and  engagement  performances 

These  parameters  reflect  the  experiments  that  will  be 
conducted  on  the  IFFN  system.  Only  the  five  parameters 
defined  above  will  be  varied  when  applying  the  SEA 
methodology:  other  parameters  describing  either  the  system  or 
the  context  will  be  fixed  in  this  analysis. 

D.  Measures  of  performance 

After  having  defined  the  parameters,  one  must  specify  the 
MOPs  of  interest  for  the  system  at  hand.  These  MOPs  must 
allow  one  to  make  a  decision  concerning  the  system:  they  must 
have  a  clear  physical  interpretation.  Let  us  denote: 

x(t):  number  of  friends  at  time  t 
xg:  initial  number  of  friends 
y(t):  number  of  enemies  at  ume  t 
yg  initial  number  of  enemies 

x(i):  number  of  neutrals  at  time  t 
Zg:  initial  number  of  neutrals 

n(t):  fraction  of  friendly  forces  lost  at  time  t 
m(i):  fraction  of  enemy  forces  at  ume  t 

n(t)-  (xg-  x(t))  /xg  .  m(t>-  (yg-  y(t))/y0  (8) 


Quantities  xg.  yg.  zg  are  measured  at  the  initial  time:  that  is  the 
tune  when  hostile  aircraft  enter  the  detection  volume  (Rg)  of  the 
system.  Quantities  x(Tf),  y(Tf),  i(Tf)  are  measured  at  the  final 
time  Tf.  The  battle  stops  when  either  the  friendly  forces  or  the 

enemy's  have  lost  a  given  fraction  of  their  assets.  The  final  time 
Tf  is  defined  by 

(n(t)<nf  and  m(t)<mf  forall  OjtsTf), 
and  (n(Tf)«nf  or  m(Tf)«mf)  (9) 

wnere  nf  and  mf  represent  the  strategy  of  each  side:  since  the 
fnendlv  forces  are  defending  their  own  territory,  they  are 
probably  willing  to  loose  a  greater  fraction  of  their  forces  than 
the  enemy:  nf  >  mf  is  very  likely  to  be  true. 

In  older  to  enable  one  10  evaluate  the  system,  the  MOPs  we 
will  consider  must  have  a  clear  physical  meaning:  as  pointed  out 
above,  we  want  the  system  to  perform  a  threefold  task,  deter 
enemy  from  entering  the  fnend  s  territory,  stop  the  enemy  as  far 
as  pouible  from  this  territory  and  before  he  can  fire  missiles 
aimed  at  friendly  aaaeu,  and  kill  u  few  neutral  as  possible. 

To  evaluate  the  first  task,  we  need  a  quantity  indicating 
whether  the  friends  win  the  battle  or  not;  as  indicator  of  the 
wilfangneta  of  the  friends  10  keep  on  fighting  is  the  redo 

a(TfV(xg*(l-nf))  ■  (l-nOfMl-nf).  (10) 


This  ratio  measures  how  far  the  remaining  forces  of  the  friends 
are  from  their  lowest  aceeptale  level  as  given  by  1-nf.  If  the 

friends  win  the  battle,  this  ratio  will  be  greater  than  one;  if  they 
loose,  it  will  be  equal  to  ooe  since  they  are  giving  up  when  their 
level  of  losses  luehes  nf.  A  similar  ratio  can  be  defined  for  the 

enemy.  Then,  we  will  consider  as  the  first  MOP  of  our  problem, 
the  ratio  of  these  two  ratios:  this  ratio  of  ratios  will  compare  the 
willingness  of  the  two  opponents  to  keep  on  fighting.  Thus  we 
define 

MOP1  -  Ix(Tf)/(x0a(l-iif))J/ly(Tfy(y0*(l-mf))]  (1 1) 

If  MOP1  >  1,  the  hostile  aircraft  fly  back  because  their  losses 
have  reached  tnf  while  n(Tf)  <  nf .  If  MOP1  <  1,  the  friendly 

forces  give  up  and  loose  the  battle  because  their  losses  have 
reached  nf  while  m(Tf)  <  mf  [Martin  1986]. 

The  second  quantity  we  want  to  evaluate  is  the  number  of 
neutral  aircraft  killed  by  the  friendly  forces;  indeed,  since  we  are 
interested  in  evaluating  the  friend's  system  as  opposed  to  the 
enemy's  one,  we  consider  only  the  neutrals  shot  down  by  the 
friends'  air-defense.  Thus,  we  are  interested  in  the  number  of 
neutrals  remaining  at  the  end  of  the  battle;  the  MOP  that  will 
measure  this  performance  of  the  system  is: 

MOP2  »  sfTfVzg  (12) 

The  last  quantity  of  interest  is  the  distance  of  the  enemy 
from  the  FSCL  when  the  battle  ends.  This  is  measured  by  the 
following  ratio: 

MOP3 -Rf/Rj  (13) 

where  Rf  is  the  distance  of  the  remaining  enemy's  aircraft  at  the 
final  time  Tf. 

If  MOP3  >  1.  the  enemy  aircraft  arc  stopped  before  they  can  fue 
missiles  aimed  at  friendly  assets.  If  MOP3  <  1,  the  enemy 
aircraft  can  fire  missiles  before  being  stopped.  The  greater  this 
ratio  is,  the  better  it  is  for  the  friendly  forces. 

E.  Mapping  from  the  parameter  space  into  the  MOP  space 

We  can  map  now  the  parameter  space  into  the  MOP  space; 
the  basis  of  this  development  is  die  work  completed  by  Logicon 
[Logicon  1986],  where  about  thirty  quantities  representing  the 
way  the  IFFN  system  performs  are  defined.  Most  of  these 
quantities  are  conditions)  probabilities  that  describe  the  different 
stages  of  the  air  defense  process.  These  quantities  are  called 
MOPs  and  MOEs  in  the  Logicon  documentation;  since  they  are 
different  from  the  MOPs  and  MOEs  we  consider  for  the  SEA 
methodology,  and  since  we  want  to  keep  the  notation  defined  by 
Logicon,  these  quantities  will  be  denoted  "Mop”  and  "Moe"  as 
opposed  to  "MOP"  and  "MOE’’  in  the  SEA  methodology.  First 
of  all,  we  are  going  to  make  assumptions  about  the  value  of 
these  Mop's  and  Moe's  in  terms  of  the  parameters  (the 
independent  variables);  then  we  will  aggregate  the  conditional 
probabilities  (Mops  and  Moes)  defined  in  the  IFFN  documents 
(Logicon  1986]  in  order  to  determine  four  basic  quantities, 
probability  of  engaging  a  friend,  a  neutral,  or  a  hostile,  and  the 
time  elapsed  between  detection  and  engagement;  finally,  we  will 
use  a  Lancbester  model  (Ekchian  1982,  Taylor  1974,  Moose 
and  Wozencrafi  1983] »  derive  the  MOPs  (for  our  problem)  in 
terms  of  the  parameters.  The  process  outlined  above  is  presented 
in  Fig.ll. 
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Fig  1 1:  Steps  in  the  determination  of  the  w*Vi 


Let  us  now  introduce  briefly  a  simplified  version  of  the  Logicon 
model.  The  basic  stages  of  the  air  defence  process  are: 
detection,  identification  (ID),  comparison  between  different  IDs 
coming  from  different  detecting  units,  conflict  resolution, 
allocation,  and  engagement.  A  conditional  probability  is 
associated  to  each  of  these  stages.  To  these  six  stages 
corresponding  to  physical  processes,  one  must  add  a  fictitious 
step  that  describes  the  probability  of  true  identification.  The 
stages  of  this  process  ate  described  in  greater  detail  in  (Martin 
1986]. 

Each  of  the  conditional  probabilities  (Moe  or  Mop)  that 
characterizes  the  basic  stages  of  die  process  is  then  expressed  in 
terms  of  the  parameters:  because  of  the  lack  of  accurate 
information,  only  rough  estimates  are  considered 

For  example,  the  probabilities  Mop3.5  (probability  of 
conflict  between  the  ID  of  two  different  sensors)  and  Mop3.6 
(probability  of  conflict  resolution)  are  assumed  to  be  functions 
of  the  level  of  centralization  only.  The  probability  of  conflict 
(Mop3.5)  must  decrease,  and  the  probability  of  conflict 
resolution  (Mop3.6)  must  increase  when  the  process  becomes 
more  centralized  (P4«l  when  centralization  is  total). 


Mop3.5  ■  0.75  -  O.S*P4  (14) 

Mop3.6 ■  0.25 + 0.5*P4  0*P4i  1  .  (13) 

The  value  of  the  other  conditional  probabilities  in  terms  of 
the  parameters  are  given  in  [Martin  1986],  Given  the  decision 
trees  that  model  the  IFFN  process,  it  is  then  possible  to 
aggregate  the  conditional  probabilities  defined  above  into  three 
basic  Quantities:  probability  to  engage  a  friend  (Moe  7),  a  neutral 
(Moe  i)  or  an  hostile  (Moe  9).  Each  of  these  trees  is  conditioned 
on  the  true  ID  of  the  aircraft;  far  example,  the  tree  represented  in 
Fig.  12  assumes  that  the  true  ID  of  the  aircraft  under 
consideration  is  ’'friend".  This  tree  yields: 

Moe7  -  Moel  .Moe3.P(ih/fd)*P(a/fi).P(e/fa).A  ( 1 6) 


(l-P(ih/fd))*(l-Mop3.5)*Mop3.6  +  Mop3.5 
. . . *Mop3.e 


+  P(ih/fd)*  ( 1  -MOP3.5)*  Mop3.6 

and  (P(ih/hd)  is  the  probability  of  identifiing  an  aircraft  as 
hostile,  given  it  is  a  friend  and  given  it  has  been  detected. 


»  •  fm  /  truamO 
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Fig  is  Decision  Tree  for  Friends 
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(17) 

y«yo  at  t-0 
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(18) 

zszq  at  t*0 

for 

0stsTf 

(19) 

Moel  and  Moe9  are  computed  in  a  similar  manner.  Since 
there  is  a  time  delay  associated  with  each  sage  of  the  process,  it 
is  also  possible  with  these  decision  trees  to  determine  the  "mean 
time  elapsed  between  detection  and  engagement''  (MoelO). 

The  final  step  to  determine  the  three  MOPs  is  the  use  of 
the  Lanchester  equations  (Ekchian  1982,  Taylor  1974,  Moose 
and  Woaencraft  1983):  from  an  initial  number  of  aircraft  of  each 
type,  we  determine  the  final  number  in  each  category  (x(Tf), 
yCTf),  ifTf))  on  the  basis  of  the  probabilities  computed  from  the 
decision  trees. 

In  the  Lanchester  model,  it  is  assumed  that  all  the  friends 
are  within  the  weapon  range  of  the  enemy,  and  that  all  the 
neutrals  and  all  the  hostiles  are  within  the  weapon  range  of  the 
friend's  units.  Since  we  are  interested  in  the  performance  of  the 
indirect  ID  process,  we  consider  losses  in  friendly  forces  to  be 
due  to  the  enemy  action,  and  to  errors  within  the  friend  ID 
process;  we  consider  losses  in  the  enemy  forces,  and  neutral 
losses  to  be  due  to  the  friend’s  fire  only:  indeed,  we  are 
interested  in  the  performance  of  the  friend's  air  defense  system 
only.  The  equations  are: 


dy/dt  *  -c*x 
dz/dt « -d*x*z 


where  "a"  is  the  probability  of  engaging  a  friend  per  unit  of  time  : 
thus, 

a  »  Moe7/Moel0 
Similarly, 

c>Moc9/MoelO  and  d*Moe8/MoelO 

b,  the  probability  for  an  enemy  to  kill  a  friend,  is  assumed  to  be 
exogenous  and  ftxed  independently  of  the  parameters. 

Equations  17, 18  and  19  can  be  easily  integrated  [Martin  1986], 
and  the  final  time  Tf  is  given  by  : 

n(Tf) » r\f  or  m(Tf)  ■  mf  (20) 

We  can  now  compute  the  three  MOPs  (MOP1,  MOP2, 
MOP3)  of  the  system  for  each  point  (Pj,  P2,  P3,  P4,  P5)  in  the 

parameter  space: 

MOP1  -  [x(TfV(x0.(l-nf))]/ly(Tf)/(y0*(l-mf)))  (21) 

MOP2  ■  z(Tf)/zQ  (22) 

MOP3  «  (Mopl  .2  -  V.Tf)/R  j  (23) 

(Mopl.2  is  the  range  from  aircraft  to  FSCL  at  time  of  detection). 
V.  APPLICATION  OF  SEA  AND  EXPERIMENT  DESIGN 

The  results  of  the  System  Effectiveness  Analysis 
methodology  applied  to  the  mathematical  model  developed  in 
section  IV  are  presented  in  this  secoon. 

A.  Mission  Locus 

In  order  to  evaluate  the  system  at  hand  vis  a  vis  the  mission 
it  has  to  perform,  the  mission  requirements  must  be  expressed  in 

terms  of  the  MOPs  defined  for  the  system. 


Qualitatively,  the  mission  the  system  has  to  fulfill  is  to 
deter  enemy  aircraft  from  invading  friendly  territory,  without 
n^UUVt'  J?  Prevcn‘  «oe»y  aircraft  from  firing 
*t5?endly  “**“• of  the  MOPs,  it  mean! 
that  MOPi  must  be  greater  than  I,  that  MOP2  must  be  as  close 

as  possible  to  1.  and  that  MOP3  must  be  greater  than  1. 

By  definition,  MOP2  is  less  than  1,  MOP3  is  less  than 
R{yRl'  and  it  can  be  shown  that  MOPI  is  less  than  l/(l-nf) 

[ManinJ986].  The  quantitative  requirements  are  assumed  to  be 

1/tl-nf)  2  MOPI  >  MOP10  -  l.l  (24) 

1  2  MOP2  2  MOP2q  -  0.8  (25) 

1  iMOP3  t  MOP30  -  1.0  (26) 

in*.  RcUti00.(24)  "***»*»  «ends  »  win  the  battle  with  a 
10%  margin;  inequality  (25)  requires  that  no  more  than  20%  of 
awl,*1*£0B rf6>  requires  the  enemy  forces  to 
be  stopped  before  they  have  crossed  the  line  from  which 

P**  Tbe*e  «qu«ments  define  in 

ISL  a  m^non k>cul *hown  inrig.  13.  One  should 

note  that  this  mission  locus  is  bounded. 


Illition  locus 
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Fig.13  Projections  of  the  Mission  locus 

For  nf-0.6,  and  the  requiremenu  set  up  above,  if  1  iu 
denotes  the  mission  locus  and  VfL,,,)  iu  volume,  one  can 
compute 

v(l-m)  ■  0.28  (27) 

B.  System  Locus 

rep7Swt  ^  ,0CUJ- w  wiO  consider  a  family  of 
partial  loci:  for  each  of  these  partial  loci,  parameters  Pj  and  P2 

will  be  held  constant,  and  parameters  P3,  P4,  P5  will  be  varied. 

If  Pj  is  held  constant,  it  means  that  the  time  delay  to  pass 

information  from  one  node  of  the  system  to  another  is  kept 
constant.  Similarly,  if  P2  is  held  constant,  it  means  that  the  Air 

Control  Procedure  is  not  changed.  Then,  the  entire  locus  is 
considered  as  a  union  of  partial  loci  -  it  allows  for  a  more 
complete  interpretation  of  the  plots. 

We  assume  the  range  of  parameter  variation  shown  in 
Table  1: 


P 

1* 

1* 

«< 


1.^ A  Va.^'i 


s: 


-7-  Wl-- 


Table  1:  Parameter  ranges 


TmiMsy  to  Warmwkxi 

o.to 

0.05 

'l 

AirCmmlProcadurt 

0.75 

0.95 

Qwkly  at  Umficauan 

0.75 

0.99 

% 

Lml  af  CMihnuon 

030 

0.99 

r. 

Qaatay  of  Q*A  Vf  *vica» 

0.75 

0.99 

These  ranges  have  been  chosen  to  yield  realistic  values  for 
the  aggregate  quantities  defined  by  Logicon  [Logicon  I986)(the 
“Mop?"  and  "Moes"  considered  in  section  IV). 

We  will  consider  four  partial  loci,  corresponding  to  the 
mwimai  and  minimal  values  of  P]  and  Pj : 


P1  *  plmax  Md  p2  *  p2max 
pl  -plmin  “d  p2  *  p2max 
P1  *plmin  ^  p2“ p2min  ' 
P1  “Plmax  *"d  p2  “ P2min 


— ->  Partial  Locus  *1 

- >  Partial  Locus  #2 

- >  Partial  Locus  #3 

- >  Partial  Locus  #4 


Before  showing  pictures  of  the  whole  locus,  let  us  set 
Pj -constant,  P2*constant,  Pj-constant  and  consider  the  set  of 
MOP  points  obtained  by  varying  P4  and  P5:  this  will  yield  a 
"slice"  of  the  partial  loci  we  will  obtain  later,  and  give  us  insight 
into  the  locus  construction;  a  typical  "slice"  is  shown  in  Fig.  14 
and  Fit  15.  Fig.14  corresponds  to  the  projection  of  this  slice  on 
the  plane  (MOP1/MOP2),  and  Fig.15  to  the  projection  on  the 
plane  (MOP1/MOP3). 

T«»*a 


Fig.14  A  slice  of  the  system  locus  in  the  plane  MOP1/MOP2 


Fig.15  A  slice  of  the  system  locus  in  the  plane  MOP1/MOP3 


The  purpose  of  these  two  figures  is  to  show  the  shape  of  a 
typical  slice  of  the  system  locus;  actual  and  accurate  plots  will  be 
shown  later.  One  can  note  an  irregularity  that  ccresponds  to 
MOP1-1,  that  is  to  the  change  in  the  tenninating  condition  (time 
Tf):  if  MOP1  is  less  than  1  the  friends  give  up;  if  MOP1  is 

greater  than  1,  the  enemies  five  up. 

In  Fig.15,  MOP3  increases  if  MOP1  is  greater  than  1  and 
if  MOP1  increases:  indeed,  the  wider  the  margin  by  which  the 
friendly  forces  are  winning,  the  farther  from  the  FSCL  line  the 
enemy  is  repulsed;  on  the  other  hand,  if  MOP1  is  less  than  1, 
MOP3  increases  as  MOP1  decreases:  indeed,  if  the  friends  are 
loosing,  the  wider  the  margin  by  which  they  are  loosing,  the 
smaller  the  tenninating  time  Tf  is;  the  smaller  the  tenninating 

time,  the  smaller  the  distance  traveled  by  the  enemy  during  the 
battle  is.  In  this  latter  case,  since  the  enemy  aircraft  art  not 
repulsed,  they  anil  eventually  invade  the  friend's  territory. 

In  Fig.14,  a  vertical  (or  kinked)  line  is  drawn  for  each 
value  of  P4,  that  is  for  each  level  of  centralization  as  represented 

by  P4  (P4-I  for  total  centralisation);  the  greater  the  value  of  P4, 
the  farther  on  the  left  of  the  diagram  the  corresponding  vertical 
line  is,  and  the  smaller  MOP1  is.  It  means  that  the  lower  the 
level  of  centralisation,  the  greater  MOP1  is,  that  is,  the  greater 
the  chances  of  winning  the  battle  are;  this  is  the  result  of  a 
trade-off  between  the  accuracy  in  the  ID  process  and  the  time 
needed  10  pcrforaa  the  identification:  die  more  accurate  the  ID  is, 
the  longer  it  takes.  It  turns  out  that  in  the  model,  the  time 
increase  in  the  ID  process  (fare  to  a  higher  level  of  centralization 
is  the  most  important  of  the  two  effects  (the  second  effect  being 
an  increased  accuracy).  For  a  given  vertical  line  (that  is  for 
P4 -constant),  the  greater  P5,  that  is  the  better  the  Question  and 

Answer  IFF  devices  are,  the  greater  MOP2  is:  it  means  that  the 
better  the  Question  and  Answer  devices,  the  greater  MOP2  is, 
and  the  smaller  the  number  of  neutral  killed  is.  These  Q&A 
devices  affect  slightly  MOP!  except  around  MOP1-1  where  the 
quality  of  these  devices  is  very  important:  around  MOP1-1,  the 
battle  can  be  won  or  lost  depending  on  the  quality  of  the  Q&A 
IFF  devices. 

For  the  next  plots,  P3  will  be  varied  with  P4  and  P5:  we 

will  obtain  as  many  slices  as  die  one  of  Figs.  14  and  15  as  values 
of  P3  considered.  One  should  recall  that  P3  represents  the 
quality  of  identification  (P3-I  for  perfect  ID  capabilities). 


In  Fig.16,  Ftg.17,  and  Fig.18  projections  on  the  plane 
(MOP1/MOP2)  are  represented;  in  Fig.16  the  projection  of 
partial  loci  #1  and  #4  (their  projections  on  the  plane 
MOP2/MOP2  are  the  same)  is  shown,  while  in  Fig.  17  the 
protection  of  partial  loci  #2  and  #3  (their  projections  on  the  plane 
MOP1/MOPZ  are  the  same)  is  shown.  For  these  two  latter  plots, 
if  all  parameters  but  P3  are  fixed,  an  increase  in  P3  yields  a 
higher  MOP1  and  a  higher  MOP2:  the  better  the  ID  capabilities 
of  the  system  the  easier  it  is  for  the  friends  to  win,  and  the 
smaller  the  number  of  neutrals  killed  by  the  system  is. 

Partial  loci  #1  and  #4  correspond  to  Pj-Pjma*,  lhiI  is  die 
longest  time  delay  to  pass  information  between  two  nodes  of  the 
system;  on  the  other  hand,  partial  loci  #2  and  #3  correspond  to 
the  shortest  time  delay  to  pass  information  between  two  nodes. 
From  these  two  loci  one  can  check  the  consistency  of  the  model: 
the  shorter  the  time  to  exchange  information  between  nodes,  the 
greater  MOP1,  and  the  greater  the  chances  of  winning  the  battle. 


I 


in  rig.  18  theprojection  of  the  enure  system  locus  on  the 
plane  MOP1/MOP2  is  shown ;  it  is  obtained  by  superposing  the 
two  previous  plots. 
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Fig. 16  Partial  Loci  #1  and  #4  projected  on  plane  MOP1/MOP2 


in  Fig.  19,  the  projection  of  the  entire  locus  on  the  plane 
MOP1/MOP3  is  shown .  The  upper  pan  corresponds  to  partial 
loci  #1  and  #2,  or  to  the  highest  quality  of  Air  Control 
Procedure  (P2  ■  P^)1.  the  lower  pan  of  Fig.19  corresponds 
to  partial  loci  *3  and  #4  and  to  a  low  quality  Air  Control 
Procedure  (ACP).The  better  the  A  CP  ,  the  greater  MOP  3: 
indeed,  with  a  good  A  CP  one  can  detea  an  enemy  aircraft  early 
and  therefore  stop  it  far  away  from  the  FSCL.  The  angle  at 
MOPl»l  corresponds  to  the  change  in  the  terminating 
conditions;  it  corresponds  to  the  irregularity  already  noted  on 
previous  plots  around  MOP1  -  1. 


Fig.19  Entire  System  Locus  projected  on  plane  MOP1/MOP3 


Fig.17  Partial  Loci  #2  and  #3  projected  on  plane  MOP1/MOP2 


of  the  cnnrt  system  locus  on  plane 
MOP3/MOP2  is  shown  in  Fig.20.  The  left  side  of  the  figure 
corresponds  to  partial  loci  #3  and  #4  (low  quality  ACP)  while 
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Fig.18  Entire  System  Locus  projected  on  plane  MOP1/MOP2 


Fig  JO  Entire  System  Locus  projected  on  plane  MOP3WOP2 


C  Measures  of  Effectiveness  tor  the  Model 

If  Lj  designates  the  system  locus  and  Lq  designates  the 
mission  locus,  and  if  V(L)  is  the  volume  of  L,  then,  to  compute 
the  effectiveness  of  the  mathematical  model  of  the  system,  one 

must  evaluate  V(L,  n  L^),  and  V(L„,)  or  V(L,)  depending  on 
the  MOE  one  is  interested  in.  Ej  measures  how  well  the  system 
capabilities  axe  used,  and  E2  measures  how  well  the  mission  is 
coveted  by  the  system. 


then,  the  panmeter  vectors  corresponding  to  the  4^  determined 
above  are 

Cl  -  [  0.205 , 0.850 . 0.990 , 0.500 , 0.904  ] 

C2  -  1 0.554 . 0.850 , 0.990 . 0.694 ,0.911] 

C3  -  [  0.204 . 0.950 , 0.990 . 0.500 , 0.900  ] 

O4  -  [  0.685 , 0.850 . 0.750 , 0.846 . 0.933  ] 

C5  ■  [  0.521 , 0.851  . 0.821 . 0.652 , 0.750  ] 

Cfi  -  1 0.873 , 0.750 , 0.874 , 0.990 , 0.990  ] 


For  the  basic  operating  point  considered  in  this  section,  we 

have 

V(L,n  L^)-  0.020  and  V(L,)-  0.068  (28) 

Therefore 

Ei- 0.292,  and  E2«  0.071  (29) 

E2  is  very  small  because  of  the  size  of  the  mission  locus  which 

takes  into  account  such  unrealistic  events  as  the  possibility  for 
the  friends  of  winning  the  battle  without  loosing  any  asset. 
Therefore,  the  degree  of  coverage  of  the  mission  locus  by  the 
system  locus  (E2)  is  very  low  . 

D.  Experiment  Design 

The  presentation  of  the  results  follows  the  same  format  as 
in  section  Ill. 

Step  1:  Detemtination  of  the  model  locus. 

This  step  has  been  completed  above. 

Step  2:  Selection  of  points  on  the  model  locus. 

As  mentioned  earlier,  we  inscribe  the  model  locus  in  a 
parallelepiped,  and  choose  the  points  of  contact  between  the 
model  locus  and  the  parallilepiped  (or  the  center  of  gravity  of 
these  points).  In  what  follows  a  row  vector  4  is  as  follow: 

4  -  [  MOP1 .  MOP2  .  MOP3  ]  (30) 

For  the  model  system  locus  obtained  above  and  shown  in 
Figs.  16  to  20.  six  points  of  contact  are  obtained: 

4<ji  -  [  1.639, 0.999, 1.203]  4d4  -  l  0.911,  0.687,  0.867] 
4<j2  •  [  1.432, 0.999, 1.081]  445  -  [  1.00, 0.61 1,  0.826] 

4d3  -  1 1-«9, 0.999,  1.403]  a<i6  “  [  1-028, 0.987, 0.610] 

The  six  vectors  obtained  above  represent  the  entire  system 
locus  as  opposed  to  any  of  its  region.  The  three  first  vectors 
correspond  to  maximum  values  for  the  MOPs  in  the  system 
locus.  The  three  last  vectors  correspond  to  minimum  values  for 
the  MOPs  in  the  system  locus 


Step  3:  Inversion  aiggnihm 

For  each  of  the  points  4^  determined  above,  we  compute  a 
parameter  value  gj  such  that  4^*  f(pj),  where  f  denotes  the 
mathematical  model  of  the  system.  If  one  notes  a  parameter 
vector  a  as  a  row  vector 

5  “  [  .  f*2  •  ^3  •  P4  •  f*5  3  (31) 


These  values  were  obtained  using  the  algorithm  described 
in  Section  m.  As  expected,  they  are  within  the  admissible  range 
of  variation  in  the  parameter  space  as  defined  by  Table  1.  Table 
2  summarizes  the  physical  significance  of  the  parameter  vectors 
obtained  using  the  inversion  algorithm. 

Table  2:  Physical  significance  of  the  parameter  vectors 


Tuaedeby 

k&SU 

jg& 

no aim 

Qatar  cf 
Mantificaian 

Level  of 

jonfcouon 

jgrrf 

Devices 

Uumm 

MOB 

®J 

SaaB 

Madam 

— 

Hifh 

4bubmb 

M0R 

*2 

— 

Madura 

Mttdnun 

Madura 

High 

MOB 

*3 

Sal 

— 

Mtataura 

Hifh 

MOB 

*4 

Hifh 

Madura 

Madam 

Hifh 

Kgh 
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MOB 

*J 

Madura 

Madura 

Madura 

Mrina 

Mnoura 

Mnma m 

MOB 

*6 

Kifh 

Miniwup 

Madura 

Mauraum 

MOP1  and  the  margin  by  which  the  battle  is  won  is 
strongly  linked  to  die  quality  of  identification  (ID):  the  maximum 
MOP1  is  obtained  for  the  maximum  quality  of  ID  and  the 
minimum  level  of  centralization,  and  the  minimum  MOP1  is 
obtained  fa-  the  minimum  quality  of  ID  and  the  highest  level  of 
centralization.  The  fact  that  an  increase  in  the  quality  of  ID 
improves  MOP1  is  easy  to  predict  The  greater  the  level  of 
centralization,  the  lower  MOP1  is:  this  is  the  result  of  a  trade-off 
outlined  earlier  between  the  increase  in  the  accuracy  of  the  ID 
process  due  to  a  higher  level  of  centralization,  and  the  increase 
in  the  time  needed  to  perform  this  ID  also  due  to  a  higher  level 
of  centralization:  it  turns  out  that  the  second  of  the  two  effects  is 
the  most  important  one,  thus  reducing  MOP1.  One  should  also 
note  that  MOP1  depends  on  the  time  delay  to  pass  information 
between  nodes:  the  smaller  this  time  delay,  the  faster  the 
response  of  the  system,  and  the  greater  MOP1  is. 

MOP2  appears  to  be  linked  to  the  quality  of  ID  and  to  the 
quality  of  the  Q&A  IFFN  devices  which  provide  local  ID 
information:  the  greater  the  quality  of  ID  and  the  better  the  local 
ID  information,  the  lower  the  number  of  neutrals  killed  by  the 
friendly  forces  is. 

MOP3  depends  mostly  on  the  Air  Control  Procedure 
(ACP),  and  on  the  time  delay  to  pass  information  between 
nodes:  the  better  the  ACP  and  the  smaller  die  time  delay  to  pass 
information,  the  greater  MOP3  is.  Indeed  with  a  good  ACP, 
one  is  able  to  detect  die  enemy  far  in  the  detection  volume,  and 
the  smaller  the  time  delay  to  pass  information,  the  faster  the 
response  of  the  system  and  die  greater  MOP3  is. 


MOPE 


Step  4:  Experimental  results. 

At  this  suge,  experiments  are  run  at  the  parameter  vectors 
determined  at  suge  3.  Since  we  cannot  run  experiment  on  the 
actual  system  for  the  purpose  of  this  paper,  a  mathematical 
model  which  is  slightly  different  from  the  one  introduced  earlier 
has  been  used.  The  pseudo-experimenul  values  obuined  by 
exercising  the  modified  model  are 

Se]  ■(  1.513, 0.999. 1.128  ]  2^4  « [  0.91 1. 0.688, 0.868  ] 
2^2  •  [  1.436, 0.999. 1.082  ]  2^5  -  1  0.997, 0.617, 0.830  ] 
^3  « { 1309, 0.999, 1.372 )  23*6  «[  1.006, 0.903, 0.565  ] 

Step  5:  Transformation  from  the  model  locus  into  the  actual 
locus. 

If  ini  is  a  point  in  the  model  locus  and  if  »*  the 
corresponding  point  in  the  actual  locus,  the  least  square 
procedure  presented  in  section  III  yields  the  following 
mnsfonnation 

&a  *  T&m)  *  +  V ,  (32) 

where  L  is  a  linear  transformation  (defined  by  a  3x3  matrix)  and 
y  a  constant  translation  vector.  For  the  example  at  hand  we  have 

r 6.831  0.046  -0.151*1  r0.1«l 

0.042  0.813  0.0008  V.  0.037  (33) 


-0.007  0.078  1.127 


Step  6:  Construction  of  the  actual  locus. 

For  each  point  in  the  parameter  locus  we  apply  A  -  Tof 


where  "o’  denotes  the  composition  of  two  functions  and  where 
"f '  stands  for  the  mathematical  function  that  maps  the  parameter 


locus  into  the  model  system  locus. 


For  the  example  at  hand,  since  the  shape  of  the  actual  locus 
in  qualiutively  the  same  as  the  one  of  the  model  locus,  only 
comparisons  of  the  two  loci  will  be  presented:  in  the  following 
plots,  the  contours  of  the  projections  of  both  the  model  and  the 
actual  locus  are  shown;  these  projections  are  done  on  the  planes 
MOP1/MOP2  (Fig.21),  MOP1/MOP3  (Fig.23),  MOP2/MOP3 
(Fig.24). 

E  Effectiveness  of  the  Actual  System 


From  the  actual  locus  of  the  nominal  system  constructed 
above,  one  can  evaluate  the  effectiveness  of  the  system.  If  LS4 
denotes  the  actual  system  locus,  then,  the  measures  of 
effectiveness  for  the  actual  system  are: 


Ej -0.300  (0.292),  E2  -  0.049  (0.071)  (34) 


The  value  of  Ej  obtained  for  the  actual  system  is  slightly 
greater  than  the  ones  obtained  for  the  model.  It  means  that  the 
capabilities  of  the  actual  system  are  better  used  than  one  could 
have  thought  by  studying  the  model  only.  On  the  other  hand, 
the  value  of  E2  is  slightly  smaller  for  the  actual  system  than  tit 
is  for  the  model:  the  degree  of  coverage  of  the  mission  is  smaller 
for  the  actual  system  than  for  the  model. 

In  this  section,  the  methodology  developed  throughout  this 
paper  has  been  applied  to  the  IFFN  system:  the  procedure  to 
evaluate  the  effectiveness  of  an  actual  system  has  been 
demonstrated. 
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VI.  CONCLUSION 

In  this  paper,  a  methodology  aimed  at  evaluating  actual  C3 
system  has  been  developed;  this  methodology  provides  a  means 
to  design  the  minimum  number  of  experiment  to  run  on  a  large 
scale  C3  system  in  order  to  evaluate  it  The  experiment  design 
procedure  as  well  as  the  evaluation  procedure  have  been  applied 
to  a  real  air  defense  system.  The  tools  presented  in  this  paper 
provide  the  system  developer  with  a  powerful  methodogy:  it 
gives  him  directions  so  as  to  which  experiments  he  should  run 
on  the  system  at  hand,  and  it  allows  him  to  evaluate  this  system 
based  on  well  designed  experiments. 

In  this  paper  a  crude  mathematical  model  of  the  air  defense 
system  has  been  introduced;  further  research  should  develop 

this  modeling  aspect  in  order  to  yield  as  accurate  models  of  C3 
systems  as  possible.  With  a  better  model  of  the  organization  at 
hand,  the  experiment  design  process  as  well  as  the  evaluation 
will  be  much  more  accurate  than  in  this  paper. 
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